Density Networks 1 Density Modelling

A density network is a neural network that maps from unobserved inputs to observable outputs. The inputs are treated as latent variables so that, for given network parameters, a non{trivial probability density is deened over the output variables. This probabilistic model can be trained by various Monte Carlo methods. The model can discover a description of the observed data in terms of an underlying latent variable space of lower dimensionality. We review results of the application of these models to toy problems with categorical and real{valued observables and to protein data. The most popular supervised neural networks, multilayer perceptrons, are well established as probabilistic models for regression and classiication, both of which are conditional modelling tasks: the input variables are assumed given, and we condition on their values when modelling the distribution over the output variables; no model of the density over input variables is constructed. Density modelling (or generative modelling), on the other hand, denotes modelling tasks in which a density over all the observable quantities is constructed. Multi{layer perceptrons have not conventionally been used to create density models (though belief networks and other neural networks such as the Boltzmann machine do deene density models). This paper discusses how one can use an multilayer perceptron as a density model. This deenition of a full probabilistic model with a multilayer perceptron may prove also useful for other interesting problems, for example, thèmissing inputs' problem (Tresp, Ahmad and Neuneier 1994). 1.1 Traditional density models A popular class of density models are mixture models, which deene the probability distribution over observables t as a sum of simple densities. These models arèlatent variable' models (Everitt 1984); each observation t